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CIERA Inquiry 3: Policy and Profession 

What is the Teacher Rating of Oral Language and Literacy (TROLL)? 
Can it provide an accurate measure of students’ literacy acquisition 
skills? 

In this article, we describe the Teacher Rating of Oral Language and Literacy 
(TROLL), an instrument that measures skills identified as critical in the New 
Standards for Speaking and Listening (Tucker & Codding, 1998). In 5-10 
minutes and without prior training, teachers can assess an individual child’s 
current standing with respect to skills that research has identified as critical 
for literacy acquisition. Skills assessed include language, reading, and writing 
abilities. TROLL has been used with over 900 low-income children. The 
instrument is reliable and has strong internal consistency. Its validity has 
been established in numerous ways; TROLL correlates significantly with 
scores on the Peabody Picture Vocabulary Test and the Early Phonemic 
Awareness Profile given to the same children by trained researchers. Fall 
TROLL judgements of the literacy skills of these same children correlate 
with fall but not spring assessments of literacy skills obtained with the Early 
Literacy Profile. TROLL also displays instructional sensitivity. 
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Teacher Rating of Oral Language and 
Literacy (TROLL): A Research-Rased 
Tool 



David K. Dickinson, Educational Development Center 

AUyssa McCabe, University of Massachusetts — Lowell ' 

Kim Sprague, Educational Development Center 

To be able to read and write, children first must develop sufficient oral lan- 
guage skills (Dickinson & McCabe, 1991; Dickinson & Tabors, 2001; Snow 
1983; Snow, Bums, & Griffin, 1998). Oral language skills blossom during the 
preschool years, which means that they are also very vulnerable and in need 
of stimulation during this time, as a number of major organizations involved 
in the education of young children have recognized. 

Because speaking and listening are so critical for literacy development in 
early childhood, the New Standards project, a program of the National Cen- 
ter on Education and the Economy (Tucker & Codding, 1998), has devel- 
oped research-based standards for speaking and listening for preschool 
through third grade. These standards complement those already developed 
for reading and writing. Speaking and listening standards include many spe- 
cific recommendations for teachers regarding beneficial habits of conversa- 
tion, useful kinds of talk such as narratives and explanations, and language 
conventions relevant to early childhood. Many programs will be concerned 
to see that their students meet such national standards. 

Anna is a quiet, attractive Hispanic girl, with long wavy black hair always 
tied back neatly into a ponytail and large brown eyes that seem to study 
everything around her. Her teacher has observed that she is shy with other 
children, reluctant to participate in group choruses, and often the last to join 
a line or in clean-up activities. Still, the teacher has never had to reprimand 
Anna. Only when the teacher sits down to fill out reports on the language 
development of each of her students does she realize Anna has a problem. 



A Can for DevelopmentaUy Appropriate Assessment 



In 1998, the International Reading Association ORA) and the National Associ- 
ation for the Education of Young Children (NAEYC) joined together to for- 
mulate a position statement regarding early literacy development. The 
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statement acknowledges the difficulty that teachers face, for example, in 
kindergarten classrooms where a five-year range in children's literacy-related 
skills is not uncommon (Riley, 1996). Estimating where each child is in 
terms of the acquisition of speaking, listening, reading, and writing skills is 
critical in order to provide developmentally appropriate instruction to all 
children in this wide range. The position statement (p.38, emphasis ours) is 
quite clear that 

throughout these critical years accurate assessment of children’s 
knowledge, skills, and dispositions in reading and writing will help 
teachers better match instruction with how and what children are 
learning. However, early reading and writing cannot simply be mea- 
sured as a set of narrowly defined skills on standardized tests. These 
measures often are not reliable or valid indicators of what children 
can do in typical practice, nor are they sensitive to language varia- 
tion, culture, or the experiences of young children. 

As if these difficulties were not enough, teachers typically have not been 
trained to evaluate children’s language development as it relates to the acqui- 
sition of literacy. In response to this need, this report presents an accessible 
means of evaluating each child in your classroom for literacy-related lan- 
guage development. Periodic evaluation of children using this tool will 
enable teachers and programs to assess and demonstrate their effectiveness. 



The Development of TROLL 



The Teacher Rating of Oral Language and Literacy (TROLL) is a rating tool 
developed by David Dickinson to provide teachers with a way to track the 
language and literacy development of children in their classrooms. It is rep- 
resented in its entirety in the Appendix. TROLL is designed to enable teach- 
ers to rate a child’s literacy-related skill and describe his or her literacy- 
related interests. Although it was developed for research purposes separate 
from the New Standards initiative, TROLL addresses all the central speaking 
and listening skills present in the New Standards. In fact, in constructing 
TROLL, the present researchers (who were also involved in developing 
these standards) drew on the same body of theory and research that was the 
foundation for Speaking and Listening from Preschool through Third 
Grade . TROLL also covers many of the early reading and writing skills teach- 
ers will want to track. Furthermore, TROLL allows teachers to track chil- 
dren’s interest in various language and literacy activities — something that no 
direct assessment tool can hope to capture. 



Using TROLL 



No formal training is required for using TROLL. This tool is designed so that 
classroom teachers can easily track the language and literacy development of 
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their students. TROLL only requires approximately 5-10 minutes for each 
child and, with a little planning, can be completed without disrupting class- 
room activities. 

Teachers can use the information to inform their teaching. First, teachers 
can identify children who are most in need of rich language and literacy 
stimulation in order to catch up to their peers in this regard. Second, teach- 
ers could combine the results for all children in their classes to determine 
which areas of performance might merit more systematic instruction for the 
entire class, or at least a group of children. For example, if all children in a 
class score relatively low on rhymes, their teacher might want to begin pro- 
viding numerous opportunities to listen to and produce rhyming chants, 
songs, and poems. Or, more likely, a teacher might find that half or two 
thirds of the class could benefit from more rhyming activity. 

School reading specialists can rate children in a classroom independently of 
the classroom teacher. Comparing ratings of individual children can serve as 
a reliability check and as an excellent, focused springboard for discussion. 
Furthermore, the reading specialist might consider using TROLL ratings of 
children as a grounding for discussion with parents. 

Parents constitute a third group of potential TROLL users. Teachers of bilin- 
gual children often have a very difficult time rating the language compe- 
tence of children who speak English as a Second Language. In such cases, 
educators are advised to enlist the help of these children’s parents. In fact, 
we found that maternal reports of preschoolers’ literacy (when children 
were three or four years old) significantly predicted much of the variation in 
kindergarten tests, first-grade teacher assessments, and direct assessments of 
decoding given near the end of first grade (Dickinson, & DeTemple, 1998). 



The Contents of TROLL 



TROLL contains three subscales: (a) language use, (b) reading, and (c) writ- 
ing. Introductory questions determine the language(s) the child speaks and 
his or her comprehension and production abilities in English. Teachers are 
given the opportunity to rate English and native language competence. The 
tool has 25 items each measured on a scale that varies slightly by item (i.e., 
most but not all use a 4-point scale). Total scores are easily calculated simply 
by adding individual scores on these 25 scales; total scores vary from a mini- 
mum of 24 to a maximum of 98. These total scores provide teachers both an 
indication of an individual child's development relative to other children and 
a means to chart an individual child's growth. 

In Table 1 we display what different scores on TROLL indicate about a child's 
overall developmental level. For example, a score of 61 indicates that the 
child is making progress that is average for four-year-olds in the fall. We have 
reported the TROLL total score that corresponds to particular percentiles by 
converting the raw scores of our total sample to percentiles. This norming 
sample was composed of low-income, high-risk children, so that these 
norms should be regarded as provisional. However, we argue that the well- 
known academic disadvantages of low socioeconomic status (SES) pre- 
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school children (e.g., Stipek & Ryan, 1997) make this norming sample 
important in its own right, especially for those who work with low-income 
children. If a child from a low SES family scores at the 10th percentile, for 
example, this result cannot be dismissed as simply due to economic disad- 
vantage; such a child is scoring very poorly relative to his or her economic 
peers. 



Table 1 : What TROLL Scores Mean 



3 year olds' 
TROLL scores 


4year olds’ 
TROLL scores 


5 year olds’ 
TROLL scores 


Relative standing 
on the TROLL 


Recommendations/Meaning 


Fall 

n-115 


Spring 

n-55 


Fall 

n=336 


Spring 

n=234 


Fall 

n=83 


Spring 

n-229 






40 


44 


43 


46 


51 


55 


10th percentile 


Assessment by child by audiologist, 
speech-language pathologist. Discuss 
Concerns with parents. 


44 


49 


52 


55 


59 


65 


25th percentile 


Assessment of child by speech-lan- 
guage pathologist, extra involvement 
in extended conversations and other 
literacy activities. 


51 


56 


6l 


66 


68 


76 


50th percentile 


Child is performing at an average 
level. 


61 


62 


71 


74 


75 


85 


75th percentile 


Child is performing above average 


68 


69 


80 


84 


85 


91 


90th percentile 


Child should be encouraged to read 
and write at advanced levels in 
school and at home. 



Psychometric Properties of TROLL 



Field Testing 



More than 900 children have been assessed over the last several years. New 
England Quality Research Center for Head Start (NEQRC) used this instru- 
ment, as did the Literacy Environment Enrichment Program (LEEP), a profes- 
sional development program designed for use with preschool teachers and 
their supervisors. 1 



Reliability 



Teachers independently rated 272 children twice, and their ratings were 
comparable. Children who scored high on TROLL the first time also scored 
high the second time they were assessed, despite the fact that teachers did 
not have access to their prior ratings. 2 




£§T COPY AVAILABLE 



TROLL 



This tool has high internal consistency, with alphas exceeding .89 for all 
ages (see Table 2). In other words, there is good reason to believe that the 
instrument is measuring a child’s general oral language acumen rather than 
separate oral language skills. Note that alphas for separate language, reading, 
and writing subscales by age ranged from .77 to .92, representing strong 
internal consistency. 



Table 2: Descriptive Statistics and Internal Consistency Measures 



Age 


n 


M 


SD 


Range 


Ajlpha* 


3 years Fall 


113 


52.64 


11.89 


29-87 


.94 


Spr 


55 


56.40 


10.19 


36-81 


.89 


4 years Fall 


336 


61.63 


13.61 


30-97 


.95 


Spr 


234 


65.13 


13.85 


29-95 


.95 


3 years Fall 


83 


67.12 


13.62 


29-96 


.94 


Spr 


229 


74.21 


13.78 


37-97 


.95 


All Ages Fall 


534 


60.55 


14.00 


29-97 


.95 


All Ages Spr 


518 


68.22 


14.70 


29-97 


.95 



•Cronbach’s alpha (Cronbach, 1951). 



Validity 



How well does this tool actually measure what it is intended to measure? 
This tool relies on a teacher’s professional judgement or perception of a 
child’s development rather than formal testing of actual development. It is 
therefore reassuring to find that the child ratings teachers provide using 
TROLL are largely consistent with those obtained by formal testing. 
Researchers administered the following formal tests to children who had 
been independently rated by their teachers using TROLL. The range of corre- 
lations between TROLL and the tests is reported in Table 3. 

1. The well-established Peabody Picture Vocabulary Test (PPVT-III; Dunn & 
Dunn, 1997) is a measure of receptive vocabulary. We used the PPVT raw 
scores because TROLL itself is a raw score. 

2. The Emergent Literacy Profile (ELP; Dickinson & Chaney, 1997) assesses 
children’s ability to read environmental print (e.g., picks out “McDonald’s”® 
from the logo) as well as children’s sense of printed language (e.g., picks the 
word out of a display such as: NNNT, W3#NJ, MILK), their ability to identify 
letters, and their early attempts at writing. This tool was developed by the 
New England Research Center on Head Start Quality. Cronbach’s alpha, a 
measure of internal consistency, equals .86 (n=578), representing strong 
consistency. 

3. The Early Phonemic Awareness Profile (EPAP; Dickinson & Chaney, 1997) 
assesses children’s ability to engage in phoneme deletion (e.g., what is “foot” 
without the “f”?) and recognize rhymes. Like the ELP, this tool was devel- 
oped by the New England Research Center on Head Start Quality. Cron- 
bach’s alpha equals .93 (n=563), representing very strong consistency. 
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Table 3: Correlations* Between TROLL Subscales and Other Measures of Language and Literacy 
Development — QRC and LEEP data— All Ages Combined (n=46 1-688). * 





TROLL — Total Score* 
n=29 1/270 


TROLL — Language 
n=304/287 


TROLL — Reading 
n=295/275 


TROLL— Writing 
n=306/278 




Fall 


Spring 


Fall 


Spring 


Fall 


Spring 


Fall 


Spring 


PPVT — Raw Score 
n=131/131 


.47*** 


.42— 


.47*** 


.40*** 


.40*** 


.38*** 


.43— 


.42— 


Emergent Literacy 
Profile— -Total 
n=68/67 


.43** 


ns 


.42** 


ns 


.29* 


ns 


.51*** 


ns 


Early Phonemic 
Awareness 
Profile— Total 
n=157/133 


.45— 


.47*** 


.34— 


.46*** 


.43*** 


.42— 


.47*** 


.44*** 



* The EPAP and PPVT were administered in English to all children, regardless of language dominance. The ELP was administered 
in the child’s dominant language, 

^ TROLL total scores fall and spring r=.88***. 

*p < 05 

**p < .01 
***p < .001 

Teacher ratings 3 of children’s language and literacy development on TROLL 
show moderate associations with children’s scores on all three formal mea- 
sures. In other words, in about five minutes and with no special training, 
teachers themselves can index what specially trained researchers would 
spend 25-30 minutes per child assessing. Of course, the TROLL teacher rat- 
ings do not agree completely with the researchers’ tests. This difference 
partly reflects the fact that TROLL captures other factors that teachers take 
into consideration as they rate individual children — factors that are not cap- 
tured in researchers’ direct, formal assessments. In other words, TROLL cap- 
tures the kind of assessment information recommended in the position 
statement by IRA and NAEYC. Formal tests measure how well a child does at 
only one point in time; children may be tired or sick on the day of the PPVT- 
III assessment and receive a dismal score for their receptive vocabulary, 
whereas their teacher knows that on most days they are quick to pick up on 
the vocabulary of classroom units and articulate when sharing stories of per- 
sonal experience. The TROLL score is not as vulnerable to fluctuations in a 
child’s performance as the formal tests. Also, as noted earlier, TROLL 
includes information about children's engagement in literacy activities and 
patterns of oral language use. 

TROLL scores correlate well with our measure of phonological awareness 
(EPAP), as is shown in Table 4. Numerous researchers have established that 
phonological awareness is a critical precursor, correlate, and predictor of 
reading achievement (Bryant, MacLean, & Bradley, 1990; Cronin & Carver, 
1998; Speece, Roth, Cooper, & de la Paz, 1999; Stanovich, 1992; Vellutino & 
Scanlon, 2001; Wagner, Torgesen, Laughon, Simmons, & Rashotte, 1993; 
Wagner et al., 1997). 

Another point worth noting is that TROLL total scores are more closely 
related to formal language scores — phonological awareness and vocabu- 
lary — than they are to literacy scores (see Table 4). Of special significance is 
the fact that, in the fall, teacher’s ratings of children’s print-related knowl- 
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edge correlate with children’s scores on the Early Literacy Profile (e.g., envi- 
ronmental print, letter identification) whereas no such relationship exists in 
the spring — the opposite of what one would predict. 

How can this be? After all, teachers have had far more interaction with chil- 
dren — and should therefore be more sensitive — in the spring than in the fall. 
One possibility is that in these preschool classrooms, throughout the year 
teachers have more occasions to observe children’s spoken language than 
their literacy activities. In the fall they may make special efforts to identify 
children’s level of literacy development, but either fail to revisit this assess- 
ment or have relatively few occasion to observe children using literacy so 
have no basis for accurately gauging their growth. Thus, children may make 
progress in their knowledge of print as registered by the ELP that goes unde- 
tected by their teachers. Alternatively, children may not be making such 
progress and teachers fail to note this. This finding of reduced ability to 
gauge children’s literacy status is sobering. The danger of any teacher judg- 
ment is that it become a self-fulfilling prophecy (Rosenthal & Jacobsen, 
1968); teachers’ opinions of children at the very outset of their education 
may predict children’s success. They may give extra attention and motivation 
to the children they expect to become the most accomplished; conversely, 
they may fail to support children they expect to fail. 

This lack of a revision of judgements about individual children’s literacy 
skills and interests is regrettable and points to the potential value of teachers 
carrying out periodic informal assessments of children to provide concrete 
evidence of children’s growth. For example, teachers can ask children to 
identify letters in their own names and in those of other children in the 
classroom, to write their names, etc. 

In short, rather than promote such self-fulfilling prophecies, we hope that 
TROLL will predict possible failure in order to prevent failure . That is, we 
hope that teachers will use this instrument to give children who are strug- 
gling the help they need so that they never have to experience true failure at 
a later point. 



TROLL Measures Effective Instruction 



Teachers often struggle to keep track of children’s growth. Researchers have 
asked teachers to track over 900 children using TROLL. In the New England 
Quality Research Center study of Head Start programs, there was a signifi- 
cant improvement in 272 children’s TROLL scores in the spring versus the 
fall (t [271]=l4.65, £=.0001). Teachers were developing children’s language 
and literacy skills, and TROLL reflected the gains made by individual chil- 
dren. In other words, TROLL displays instructional sensitivity. 

TROLL has also been able to detect changes that occurred as a result of a 
program improvement effort. Head Start teachers and their supervisors vol- 
unteered to participate in the LEEP program (previously introduced). They 
received academic credit for participating in two intensive three-day blocks 
three months apart. TROLL scores for children whose teachers participated 
in LEEP were significantly higher than for children whose teachers did not. 
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Specifically, children in LEEP classrooms gained more overall from fall to 
spring on average, compared to a control group (F=9.24, p=.003). 

Furthermore, the classrooms that supported such advances were indepen- 
dently observed to have improved classroom language and literacy practices. 
Teachers who participated in LEEP made a significantly greater effort to 
engage children in conversation and to provide opportunities for children to 
write and use books. The most major shift was represented by the extent to 
which teachers planned activities with the intention of having children prac- 
tice literacy-related skills. Enriching the literacy environment had one addi- 
tional surprising impact: Children whose teachers participated in LEEP 
displayed significant growth in social skills 4 as compared with their peers 
(Fn=8.46, pn=.004). Children who are busy talking, reading, and writing- 
activities registered by TROLL — were more likely to be viewed by their 
teachers as developing stronger collaboration skills. 



A Final Note 



We conclude by returning to Anna’s story. Anna scored the lowest on TROLL 
of anyone in her class — a total of 40 points. Her teacher realized that she 
spent far more time talking to Anna’s high-scoring classmates than she did to 
Anna. Children who were already the most advanced talkers were the ones 
who asked her for the names of things that interested them, participated in 
group discussions, and took many opportunities to explain activities to 
other children or tell stories about themselves. In other words, the teacher 
found that in her classroom, ak in the classrooms of other preschool 
teachers 5 , the old adage “The rich get richer” applied to language and liter- 
acy development. The teacher also realized that several of the children who 
acted out frequently also received low scores, and she made an effort to 
involve them in more conversations. Two such boys, in particular, seemed to 
thrive on this extra attention for positive behavior, and were noticeably bet- 
ter behaved by the end of the year. 

In the subsequent weeks, Anna’s teacher made a concentrated effort to 
involve her in talk every day. Anna looked at her politely during these initia- 
tives, but she seldom responded. Anna was still very, very quiet. On her 
teacher’s recommendation, Anna’s parents had her hearing acuity evaluated 
by an audiologist, who discovered that Anna had a mild hearing loss. Anna, 
like many other children (Friel-Patti & Finitzo, 1990), had previously unde- 
tected hearing loss that made it hard for her to learn in class. With medical 
treatment, Anna’s hearing problem was readily resolved. However, she still 
struggled with language learning and delay and benefited from speech and 
language remedial services. Because her hearing loss was discovered so 
early, her parents, teacher, and speech language pathologist believe she^has 
every chance to speak, read, and write with her peers. She did not have to 
fail at reading in order to get the help she needed to succeed. Anna benefited 
from the kind of early intervention strongly recommended by Snow, Bums, 
and Griffin (1998, pp. 318-319). 
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Notes 



1. For more information about LEEP contact David Dickinson at EDC, 55 
Chapel St., Newton, MA 02458. 

2. It was not feasible to obtain test-retest reliability estimates, as teachers 
would have had to rate the same children twice within a short amount of 
time, a request that would have been unreasonable. Also, a close retest 
might have allowed examiner memory (and hence bias) to compromise the 
second rating. 

3. There were several small changes in the technical details of the TROLL 
making it inappropriate to conduct psychometric assessments using the 
sample of 973. Hence we reduced the sample size. Analyses with the earlier 
sample had similar patterns. 

4. This finding came from teacher ratings using an adaptation of another 
tool, the Social Skills Rating System (Gresham & Elliott, 1990). 

5. See the report of patterns, of classroom interaction in preschool class- 
rooms in Dickinson and Tabors (2001). 
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